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Three players bring their avatars to the same in-game location to start a quest together. As they arrive, the 
gnome bounces on the spot and waves. The elf throws back her head in laughter before dancing with a 
provocative hip sway. The human salutes smartly and bows. Each of these gestures communicates information 
about the avatars themselves and about the interaction choices of the players controlling them. 


An avatar’s movement includes a range of motions, from programmed gaits and postures to the ways in which 
the avatar moves in and through virtual space as guided by the player. Gestures are a subset of this movement -- 
a specific kind of motion that encodes personal, social, and cultural information. Gestures can be decoded by 
others who share an understanding of the relevant codes, communicating information about intentions, 
emotions, and responses to events and to other people. Gesture and movement can play a key role in avatar- 
mediated relationships—both in interactions with other players and in information exchanges between players 
and their own avatars. The importance of these conveyances may be intensified in immersive gaming as an 
avatar’s digital body becomes more closely aligned with a player's physical body, with the potential for influences 
on the player’s embodied experience. 


Gesturing to Others 


As humans interact in physical spaces, there are inherent tensions between gestures as “signifiers for meanings” 
(linguistic indicators) and gestures as “indexical of subjectivity and presence” (evidence of the presence and 
intentionality of an individual subject) (Noland, 2008, p. xii). In other words, there is a disjoint between our 
movements as expressions of ourselves as subjective beings and what those movements mean in terms of a 
specific socio-culturally agreed gestural language—one might, for instance, ball a fist in an expression of internal 
tension, but that may be interpreted by others as aggressive. Videogames cause such tensions to be heightened as 
the subjective being is the player, but most of the gestural content belongs to the avatar and the game. As with 
other avatar features (see Ahn, this volume; Fox, this volume, Nowak, this volume), avatar gestures tend to be 
predefined by programmed mechanics, creating limited palettes of socio-culturally agreed-upon meanings that 
promote in-game communication norms. Players choose gestures from those palettes by typing words or phrases 
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known as “emotes” (e.g. /bow, /salute, /cheer), and these commands cause the avatar to perform the designated 
gestures. These words describe the gestures but may also emphasize their linguistic, semiotic roles as meaning- 
signifiers. In many games, several different types of avatars may perform the same basic gesture through the 
same mechanism (e.g. typing /wave). However, the movement alters qualitatively in terms of shape and effort for 
different races or species—an Elf may wave in a large, high, smooth, open movement, whereas a Goblin’s wave 
may be abrupt, constrained, and closer to the body—but the predefined gesture retains its underlying form and 
thereby its signification of meaning (Barthes, 1964; Elam, 1980). In most massively multiplayer online (MMO) 
games, this common gestural grammar remains constant across all races. For example, World of Warcraft (WoW) 
players from different factions (Alliance and Horde) cannot speak to each other in-game since the game's design 
constrains it — their text-chat displays as nonsense to members of the opposite faction. However, they can use 
gestures to communicate, since many gestures are universally understood and since an enacted emotion also 
appears as descriptive text in the game’s chat window. Yet in everyday human life, not all gestural meanings are 
shared across different cultures (Brosnahan & Okada, 1990; Creider, 1986; Morris, Collett, Marsh & 
O'Shaughnessy, 1979). Even the apparently simple action of nodding or shaking one’s head may vary in cultural 
translation. This caused issues when Final Fantasy X-2 (2003) was translated into English. In the original 
Japanese version, the protagonist Yuna finds a sphere that belongs to a rival group and is asked whether she will 
return it. She nods to indicate, “You are right, I will not return it.” However, within the spoken context an 
English-speaking audience was deemed likely to interpret the gesture as, “Yes, I will return it? The standard 
route for audio-visual translation would be to adjust the dubbed text/vocals to ask a different question to which a 
nod is the appropriate response. However, on this occasion the company decided to change the gesture. Thus, in 
the English version, Yuna shakes her head to indicate that she will not return the sphere, rather than nodding 
(O'Hagan & Mangiron, 2013). 


As exemplified by the nodding dilemma, research in linguistics has shown that speech and gesture are tightly 
bound together in effective communication (Kita, 2009). “We gesture in order to think” (Ventrella, 2014, p. 349), 
such that people gesture even when they cannot be seen, as when talking on the telephone (Bavelas et al., 2007). 
Even people who have been blind from birth and have not seen others gesturing during speech will often gesture 
when speaking (Iverson et al., 2000). In computer games, many non-player characters (NPCs) are programmed 
to gesture while speaking, and sometimes avatars will gesture independently when interacting with others, 
according to their programmed design. Yet when MMO players are communicating with other players in text- 
chat or via headsets, their avatars usually will not gesture automatically, in tandem. (One exception is the virtual 
world environment Second Life (est. 2003), which offers typing or speaking animations when users are typing.) 
Given the deep connection between speech and gesture, it is not surprising that players tend to flavor 
conversations with their own choice of avatar gestures, although the manner of doing so varies. Some employ the 
predefined emotes, using the forward-slash key and the appropriate word; in games like Little Big Planet (2008) a 
controller joystick might drive gestures, or in Second Life gestures may been driven by an interface control panel. 
Regardless of control method, when one player makes an emote gesture, other players will often join in, so ifa 
player types /cheer it can lead to a group of avatars cheering together (Newon, 2011). However, some emotes are 
not accompanied by avatar movement, including most gestures that would involve physical contact with another. 
In WoW, typing the /hug emote brings up text describing the action and the target but the avatar does not move. 
The giver and receiver of the hug can still acknowledge the linguistic “gesture,” together with those nearby, as its 
mere description signifies an agreed cultural concept (the social hug), but if the hug is unwanted then it can be 
ignored. 


Sometimes players supplement limited options by defining their own expressive gestures from within the game's 
parameters. For instance, MMO players will sometimes greet fellow players’ avatars by bowing and casting 
unnecessary healing spells on them, such that the player’s “own cognitive and button-pushing abilities were 
combined with the avatar’s in-world abilities to jointly perform an intended friendly act” (Banks, 2015, para. 41). 
Communication theorist John Fiske describes this type of redundant messaging (i.e., unnecessary heals) as 


keeping existing channels “open and useable,” like shaking hands (1990, p.14). It is also common in MMOs for 
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groups of players waiting for something to happen to begin dancing with each other. The act of dancing is 
triggered by typing the /dance emote, making the avatar perform a set dance sequence specific to its race and/or 
gender. No further action is required by the player for the dancing to continue, and indeed a player can leave the 
computer for a short while and the avatar will continue to dance. Despite this, dancing in groups seems to be 
used as a way of indicating presence while waiting, likely because it simulates a social bond of shared activity 
that keeps “existing channels” open. In these exchanges, gesture shifts away from being a “signifier of meaning” 
and toward being “indexical of subjectivity and presence” (Noland, 2008, p. xii) - an indication that a person is 
present and available for interaction. 


Gesturing to Oneself 


In phenomenological terms, according to philosopher Carrie Noland, gestures and movements “are not only 
productive of communication between agents, they also provide the individual agent with a private somatic 
experience of his or her own moving body” (2008, p. xi). While one might not perceive the movement of 
someone else's avatar as being expressive at an embodied level, the relationship between the player and her own 
avatar is another matter. The physical movement of the player is encoded via the control mechanism (e.g., 
pressing the keyboard, moving the joystick) and decoded into the digital movement of the avatar, so that the 
player experiences correlations between her own movements and those of her avatar (known as natural 
mapping), or between her intentions and her avatar’s actions (e.g. typing the emote /bow). Natural (imitative) 
movement mapping control mechanisms (such as using a steeringwheel controller for driving games or a guitar 
for music simulations) have been associated with increased presence in virtual worlds - a stronger sense of being 
in the game environment (Skalski et al., 2011; Jenson & Castell, 2009). 


In MMOs, jumping is an intuitive example of movement mapping. Pressing and releasing a thumb on the space 
bar correlates with physically bending and releasing the knees for a jump, creating a somatic link between player 
and avatar. It could be argued that this kind of direct mapping can be just as strong as a full natural movement 
mapping mechanism when it is learned as part of a player’s embodied experience of gaming (cf. Rogers, 
Bowman, & Oliver, 2015). 


Jumping is often used to indicate presence to other players, or to gain attention from them, or simply to show 
excitement or impatience (Newon, 2011; Ventrella, 2014). However, it is also not unusual to see lone players 
jumping outside MMO dungeons and raids, where the movement may be a way of maintaining a sense of 
embodied connection and presence in/with their avatars while waiting for other players to arrive; as such, there 
may be more to effective movement mapping than simple imitation of real-world actions. 


Mapping Gestural Qualities 


The interaction between player and avatar movement is particularly well-exemplified in the experiences of 
amateur sportsman Peter Gray during his training for the Brain-Computer Interface Race in the 2016 Cybathlon 
games in Switzerland (personal communication, June 24, 2016). Gray is paralyzed from the shoulders downward 
and trained for two years to race a digital avatar using only his thoughts to control it. A close-fitting cap held 
electrodes against his head to pick up tiny electrical signals from his brain as he focused on a particular thought. 
It was against the games’ rules to move physically, so Gray thought hard about reaching for a bar of chocolate 
with his left hand or punching someone with his right fist. He rehearsed these “thought gestures” independently 
of the avatar for the first few months, so that the technical team could work on picking up the signals. However, 
when he came to control the avatar in the game (comparable to that in the smartphone game Temple Run 
[2011]), he found that the technical team had mapped his reaching thought to the avatar’s jumping movement, 
and his punching thought with the avatar sliding under a barrier. Despite his physical paralysis, this mapping felt 
wrong to Gray. His thought gestures did not match the qualities of the avatar’s movements. The extended, even 
gesture of reaching for the chocolate was inconsistent with the avatar’s sharp, repeated movements when 
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jumping, while the tense, fast gesture of punching did not equate to the avatar’s smooth sliding. Eventually the 
technical team had to swap the inputs so that Gray could control the avatar’s movements with thought gestures 
that he experienced as being naturally somatically connected. 


Gray’s experience demonstrates the importance of the gestural communication channel between avatar and 
player. The somatic inconsistency between his gestures and those of his avatar was sufficient to confuse the 
signals passing along that channel. Sometimes players perceive avatars as connecting to their body schema in the 
form of “faux body parts and physical tools” (Ratan, 2013, p. 325; see also Ratan, this volume), but Gray’s 
account suggests that he experienced his relationship with the avatar in a different way. Instead, he appears to 
have engaged his body image as a primarily conceptual sense of identity: the body, whole or part, “is recognized 
as ‘my’ body rather than an alien object” (Gallagher, 1986, p.545). While playing computer games, many people 
switch freely between referring to their avatar as “me” (“I am doing this”) and as “it/her/him” (“my avatar is 
doing this”), thus appearing to shift between incorporating and excluding the avatar in their body image. 
Although this may seem like a sense of faux body identity, according to Gallagher, body schema “is a non- 
conscious performance of the body” (p. 548), applied to “the lived physiology, but also to the way the body lives 
its environment” (p. 549). The lack of correlation between Gray’s thought gestures and the avatar’s movements 
directed his consciousness to the point of disjuncture between his body schema and the avatar. Correcting that 
correlation by creating somatically consistent mapping (but not natural mapping, formally) strengthened the 
communication channel between him and his avatar, resulting in a sense of shared movement, rather than of 
owned-but-distinct avatar body. This critical distinction between body image and body schema highlights the 
importance of movement and gesture in the connection between player and avatar, where they function as 
carriers of both identity and lived embodied experience (see also Lamerichs, this volume). 


Embodied Identity 


Although avatars’ gestures and movements may diverge from those of their players, the recent rise in augmented 
reality and virtual reality (AR/VR) gaming has shifted avatar engagement well beyond the traditional physical/ 
digital body relation. Players are now perceptually injecting their bodies into digital environments (e.g., Batman 
Arkham VR, 2016), and digital characters are appearing in physical-world spaces (e.g., Pokémon GO, 2015). So 
how do these dynamics change the meaning and function of movement and gesture? What happens if the avatar 
looks identical to the player in an augmented reality or telematic performance, or if the avatar’s and player’s 
bodies appear to share the same corporeal space in immersive VR? In these cases, the mapping of physical 
gesture and movement to the avatar is more direct, creating a potentially more integrated relationship between 
player and avatar (see Popat, 2016). 


Movement researcher Susan Kozel performed in Paul Sermon’s 1994 installation Telematic Dreaming, and 
subsequently wrote a detailed phenomenological study exploring the experiences of her “electric body” (Kozel, 
1994, p. 13). Her account reveals how her sense of embodiment fused with her avatar to enable presence with 
other people in a remote location. 


While not billed as a game, per se, the installation was deeply playful and raises issues that are likely to apply to 
immersive gaming in the future. 


Telematic Dreaming used ISDN lines to connect two rooms, each containing a double bed. Kozel was on Bed A 
with an overhead camera recording her. This camera-feed was projected in real-time onto Bed B, which was 
situated in an art gallery. Visitors to the gallery saw Kozel’s image (or avatar) as a full-scale, two-dimensional 
projection on Bed B. They could approach the bed and get onto it if they wished. Another camera, positioned 
above Bed B, recorded Kozel’s projected image combined with anyone who got onto the bed. This camerafeed 
was displayed on monitors around both beds, so that Kozel and the gallery visitors could see themselves together 
on the monitor screens. Kozel (1994; 2007) described how she worked through her projected avatar to interact 
with gallery visitors. This often began with tentative gestures, making connections between her projected fingers 
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and their physical fingers. One man moved on the bed with Kozel’s avatar on several occasions during a day. He 
returned later to place a rose upon the pillow next to her virtual head—she caressed its projected outline, but 
could not pick up. Kozel reported increasingly experiencing movement, gesture, and touch through extended 
embodiment, as if her body and her avatar developed a kind of connective membrane. At one point someone 
placed a hand upon her virtual thigh, so she moved her own hand to the same place and was surprised to 
encounter the bulk of her own physical leg (Kozel, 2007, p.100). Another person punched Kozel’s image in the 
stomach causing her instinctively to fold up as if hit, even though she could not feel the impact. 


Gabriella Giannachi discusses Telematic Dreaming in her book on Virtual Theatres, describing the point of 
connection between body and avatar as the “hypersurface.” She identifies what she calls “contamination” across 
the hypersurface between virtual and physical, where the two seem to bleed together (2004, p. 99). She notes that 
both Kozel and Sermon described some disorientation when leaving the installation after prolonged periods of 
interaction, as they struggled to re-orient themselves in their physical bodies without their avatar extensions. The 
deep correlation of gesture and movement between body and avatar had allowed their body schemas to become 
entangled with that of their avatars, in a manner similar to but more extended than Gray’s Cybathlon training 
experiences. As bodies and avatars shared correlating movement and gesture, so there appeared to be some 
“bleeding together” of embodied experience and identity. The experiences of Gray, Kozel, and Sermon have 
implications for VR gaming, where prolonged participation in virtual environments could result in experiences 
of extended embodiment (pre-conscious, and thus different from the personal identity associations that Ratan 
[2013] suggests), or conversely it could affect the experience of one’s physical body when exiting the game. 


Corporeality and Immersion 


While Kozel’s full-scale replication of the physical body in AR provides a visually representative avatar, the 
increasing use of VR head-mounted display units provides another kind of experience. In many computer games 
and simulations using VR headsets, the user’s physical body and the avatar’s digital body appear to the user to 
share the same space. Movements of one’s own head are matched with corresponding changes in perspective on 
the virtual environment, to create a “response as if real” (Slater et al., 2009, p. 294). One might xperience scaling 
a rock face (The Climb, 2016) or controlling a fighter ship in outer space (EVE: Valkyrie, 2016) almost as if one 
were actually, physically there. This apparent co-location changes the relationship between physical and digital 
bodies—and their movements—particularly since there is often no visual representation of the avatar in VR 
games, simulations, and installations, as they tend to engage a first-person perspective. (The Climb, for instance, 
displays avatar hands, but no other sense of the digital body.) Thus, the movements of the player’s body can seem 
still more closely aligned to those of the avatar. In her VR installation In My Shoes: Dancing with Myself, Jane 
Gauntlett uses simple techniques to encourage the participant to experience the avatar as if it were her own 
body, in order to convey something of what Gauntlett feels in the build-up to an epileptic seizure. The 
installation space includes physical objects, which the participant is instructed to touch while seeing the avatar 
touch similar virtual objects. The combination of physical and digital touch and gesture is thought to bind the 
participants movement to the avatar’s, increasing the sense of empathy and simultaneous physical/virtual 
corporeality. 


Perceptions of avatar and user’s bodies bleed into each other through shared movement, gesture, and bodily 
experience, so that it becomes difficult for the user to perceive what is physical or digital. 


Through self- and other-directed gestures, through extensions of identity and presence, and through potentials 
for stronger immersion and co-presence, gesture and movement play a critical role in relations between players 
and avatars, and among players via their avatars. They are also essential to the communication channel between 
the player and her avatar, with somatic exchange enhancing potential senses of “being” the avatar. Active 
correlation between movements of player and avatar can lead to an extended body schema, and this will become 
more critical as VR gaming develops. Enhancement of the lived experience of gaming through bodily integration 
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with the avatar via gesture and movement may lead to entanglement and permeability between physical and 
digital. This is deeply immersive but potentially also affects the player's lived bodily experience during and after 
the game session (Giannachi, 2004). Where such permeability occurs, the intensive nature of embodied 
experience in immersive virtual environments might have significant ethical implications for VR gaming in the 
future, particularly in the case of violent or emotionally challenging content (Popat, 2016). How closely linked 
can the bodily experiences of avatar and player become before we begin to see gamers with post-traumatic stress 
disorder? 


Throughout this discussion, the critical factor has not been whether the player claims the avatar’s gestures as 
one’s own, but how far one experiences shared movement with an avatar. In the lived embodied experience of 
moving, player and avatar may become sufficiently entangled for conceptual distinctions between physical and 
virtual to blur. Whether it is achieved via gestural communication, natural movement mapping or somatic 
correlation, sharing movement with an avatar is one of the key ways in which a player gains subjective presence 
in the gameworld. ! 
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